Lecture 13

Causal Inference in Conditionally Randomized Experiments

An Example

Researchers hypothesize that MHT increases bone density.
We’ll think about how to look into this.
Above, we’ve got a population of 10K people eligible for some form of MHT.
- Their age at the time the study begins is on the \(x\)-axis.
- Bone density, measured 2 years later, is on the \(y\)-axis.
- Color indicates whether they did or didn’t do MHT.
What we want to know is how well MHT works, on average, for the people who take it.
That means comparing green folks—people who take MHT—to versions of themselves who didn’t.

Counterfactuals

Real Population: Green People Did MHT, Red People Didn’t

Counterfactual Population: Green People Didn’t Do MHT, Red People Did

We can’t observe these counterfactual versions of people.
But we can think about what the data would look like if we did.
- On the left, we’ve got our real population.
- On the right, we’ve got a hypothetical one where everyone who did MHT didn’t and vice versa.
- Note we’re keeping the colors the same—counterfactual green people are still green.
We can think of the average difference as the average effect of MHT for those who took it.

\[ \begin{aligned} \frac{\sum_{j:w_j=1} y_j(1) - y_j(0)}{\sum_{j:w_j=1} 1} = \frac{\sum_{j:w_j=1} y_j}{\sum_{j:w_j=1} 1} - \textcolor{blue}{\frac{\sum_{j:w_j=1} y_j(0)}{\sum_{j:w_j=1} 1}} \end{aligned} \]

We call this the average treatment effect on the treated (ATT). It’s a difference of two means.
One is observable (if we have the population). The other isn’t.

Counterfactuals

Real Population: Green People Did MHT, Red People Didn’t

Counterfactual Population: Green People Didn’t Do MHT, Red People Did

We can’t observe these counterfactual versions of people.
But we can think about what the data would look like if we did.
- On the left, we’ve got our real population.
- On the right, we’ve got a hypothetical one where everyone who did MHT didn’t and vice versa.
- Note we’re keeping the colors the same—counterfactual green people are still green.
We can think of the average difference as the average effect of MHT for those who took it.

We call this the average treatment effect on the treated (ATT). It’s a difference of two means.
One is observable (if we have the population). The other isn’t.

Summarizing these Populations

Real Population: Green People Did MHT, Red People Didn’t

Counterfactual Population: Green People Didn’t Do MHT, Red People Did

The scatter plot is hard to read. To help, I’ve overlaid some summaries.
- Mean bone density for each age and treatment group.
- Histograms of age for each treatment group.
We can think of what we’re doing in terms of these summaries. Here’s the observable mean.

\[ \begin{aligned} \text{observable mean} = \frac{\sum_{j:w_j=1} y_j}{\sum_{j:w_j=1} 1} = \sum_x \frac{\sum_{j:w_j=1, x_j=x} 1}{\sum_{j:w_j=1} 1} \frac{\sum_{j:w_j=1, x_j=x} y_j}{\sum_{j:w_j=1, x_j=x} 1} = \sum_x \textcolor[RGB]{0,191,196}{p(1,x)}\textcolor[RGB]{0,191,196}{\mu(1,x)} \end{aligned} \]

It’s the average of the green curve over the distribution of the green points—i.e., the green histogram.
The counterfactual mean is the same thing, but in the counterfactual plot.

Working With Summaries

Real Population: Green People Did MHT, Red People Didn’t

Counterfactual Population: Green People Didn’t Do MHT, Red People Did

This seems like it’d be a problem, since we don’t actually see the counterfactual plot.
But, comparing our plots, we see something helpful.
- The green curve on the right looks like the red curve on the left.
- To make this easier to see, I’ve drawn it in as a dashed line.
We can use what actually happened to the red people to estimate what would have happened to the green people.

\[ \begin{aligned} \text{observable mean} &= \sum_x \textcolor[RGB]{0,191,196}{p(1,x)}\textcolor[RGB]{0,191,196}{\mu(1,x)} \\ \text{counterfactual mean} &= \sum_x \textcolor[RGB]{0,191,196}{p(1,x)}\textcolor[RGB]{248,118,109}{\mu(0,x)} \end{aligned} \]

In real data, there’s no counterfactual plot, so we can’t do this comparison visually.
But we can prove that these curves look the same in conditionally randomized experiments.

Sampling

The other part of our problem is the statistical element.
- In reality, we don’t have the whole population.
- We just have a sample of, say, 5K people.
So we have to ask whether what we see in the sample is true of the population.

Plan

We’re going to talk about how to answer these questions.
You’ve seen the main ideas before in simpler contexts.
All we’ve got to do is generalize them and put them together.

Conditionally Randomized Experiments

The Idea

To make things simple, let’s start by looking at a subset of our population.
- We’ll look at participants who were either 55 or 75 when the study began.
- We’ll ignore the others. This gives us two age groups—a binary covariate.
Suppose that each person flips a coin to determine whether they do MHT.
- The coins are weighted differently are different age groups.
- That’s why different proportions of people do MHT in the two groups.
We call this a conditionally randomized experiment.
- a.k.a. a blocked or stratified randomized experiment.

Randomization and Sampling

We start with two potential outcomes for each person in our population.
- Those are the connected dots we see in the plot.

Each one flips a coin — one with weight depending on their age — to determine whether they’re treated.
- The winning outcomes — the ones that actually happen — are the ones that don’t fade out.
- This is, of course, random. So different this happen each time we do it.

Then we sample — with replacement — from our population.
- ⦻ marks a person who isn’t sampled.
- small dots mark people sampled once.
- bigger dots mark people sampled 2,3,… times.
- This is random, too.

What Happens to Within-Group Means

We start with two potential outcomes for each person in our population.
- Those are the connected dots we see in the plot.
- At each level of \(X\), these potential outcomes have a mean.

Each one flips a coin — one with weight depending on their age — to determine whether they’re treated.
- The winning outcomes — the ones that actually happen — are the ones that don’t fade out.
- These have a mean, too. And it is, of course, random. It changes if our population flips again.

Then we sample — with replacement — from our population.
- Dot size shows how many times a person is sampled. ⦻ means 0 times.
- Our sample has a mean within in group, too. A random one.

How different are these means?

Not very different. Why?

What Happens to (Not Within-Group) Means

We start with two potential outcomes for each person in our population.
- Those are the connected dots we see in the plot.
- These have a mean outright—ignoring \(X\). Often, that’s what we’re interested in.

Each one flips a coin — one with weight depending on their age — to determine whether they’re treated.
- The winning outcomes — the ones that actually happen — are the ones that don’t fade out.
- We can look at the means of people who actually flip ‘heads’ and ‘tails’, too. It’s random.

Then we sample — with replacement — from our population.
- Dot size shows how many times a person is sampled. ⦻ means 0 times.
- And we can look a the means of sampled people who flip ‘heads’ and ‘tails’. That’s random, too.

How different are these means?

The counterfactual and population means are very different. Why?

Doing Calculations (Older not-really-integrated stuff)

\[ \begin{array}{cccc|ccc} \hline 1 & \textcolor{blue}{60} &1 & 70 & \textcolor{blue}{70} & \textcolor[RGB]{255,0,0}{70} & \textcolor[RGB]{255,0,0}{0} \\ 2 & \textcolor{magenta}{70} &1 & 80 & \textcolor{magenta}{80} & \textcolor[RGB]{255,0,0}{70} & \textcolor[RGB]{255,0,0}{10} \\ 3 & \textcolor{blue}{60} & 0 & 60 & \textcolor[RGB]{255,0,0}{60} & \textcolor{blue}{60} & \textcolor[RGB]{255,0,0}{0}\\ 4 & \textcolor{magenta}{70} &0& 70 & \textcolor[RGB]{255,0,0}{80} & \textcolor{magenta}{70} & \textcolor[RGB]{255,0,0}{10} \\ 5 & \textcolor{blue}{60} & 0&70 & \textcolor[RGB]{255,0,0}{70} & \textcolor{blue}{70} & \textcolor[RGB]{255,0,0}{0} \\ 6 & \textcolor{blue}{60}& 0 & 80 & \textcolor[RGB]{255,0,0}{80} & \textcolor{blue}{80} & \textcolor[RGB]{255,0,0}{0} \\ \end{array} \]

For each age group, we have a randomized experiment.
Among those with \(\textcolor{blue}{X=60}\) and \(\textcolor{magenta}{X=70}\) …
- Everything we’ve said before about randomized experiments is true.
- We can think of the people who do and don’t do MHT as being drawn from the same population.
- That means that — if we ignore \(W_i\) and \(Y_i\) — they have the same distribution.
- We have to ignore this stuff to get idenitical distribution because it’s downstream of randomization.
This allows us to get at the distribution of our missing data.
Remember red numbers are not observed.

Filling in Missing Data

\[ \begin{array}{cccc|ccc} i & X_i & W_i & Y_i & Y_i(1) & Y_i(0) & Y_i(1)-Y_i(0) \\ \hline 1 & \textcolor{blue}{60} &1 & 70 & \textcolor{blue}{70} & \textcolor[RGB]{255,0,0}{70} & \textcolor[RGB]{255,0,0}{0} \\ 2 & \textcolor{magenta}{70} &1 & 80 & \textcolor{magenta}{80} & \textcolor[RGB]{255,0,0}{70} & \textcolor[RGB]{255,0,0}{10} \\ 3 & \textcolor{blue}{60} & 0 & 60 & \textcolor[RGB]{255,0,0}{60} & \textcolor{blue}{60} & \textcolor[RGB]{255,0,0}{0}\\ 4 & \textcolor{magenta}{70} &0& 70 & \textcolor[RGB]{255,0,0}{80} & \textcolor{magenta}{70} & \textcolor[RGB]{255,0,0}{10} \\ 5 & \textcolor{blue}{60} & 0&70 & \textcolor[RGB]{255,0,0}{70} & \textcolor{blue}{70} & \textcolor[RGB]{255,0,0}{0} \\ 6 & \textcolor{blue}{60}& 0 & 80 & \textcolor[RGB]{255,0,0}{80} & \textcolor{blue}{80} & \textcolor[RGB]{255,0,0}{0} \\ \end{array} \]

What is the average \(Y_i(1)\) among the participants with \(X=60\)?

\[ \frac{\textcolor{blue}{70} +\textcolor[RGB]{255,0,0}{60} +\textcolor[RGB]{255,0,0}{70} + \textcolor[RGB]{255,0,0}{80}}{\textcolor{blue}{1} + \textcolor[RGB]{255,0,0}{3}} = 70 \]

What is the average \(Y_i(1)\) among the treated (\(D=1\)) participants with \(X=60\)?

\[ \frac{\textcolor{blue}{70} }{\textcolor{blue}{1}} = 70 \]

What is the average \(Y_i(0)\) among the participants with \(X=60\)?

\[ \frac{\textcolor{red}{70} +\textcolor{blue}{60} +\textcolor{blue}{70}+\textcolor{blue}{80}}{4} = 70 \]

What is the average \(Y_i(0)\) among the untreated (\(D=0\)) participants with \(X=60\)?

\[ \frac{\textcolor{blue}{60} +\textcolor{blue}{70}+\textcolor{blue}{80}}{3} = 70 \]

Conditional ATE for \(\textcolor{blue}{X=60}\)

Because we have a randomized experiment within the \(\textcolor{blue}{X=60}\) group …
The difference in subpopulation means is equal to the average effect of treatment within that group.
We call this the conditional average treatment effect (\(CATE(60)\)). \[ CATE(60) = \mathop{\mathrm{E}}[Y_i(1) - Y_i(0) \mid X_i=0] = \mathop{\mathrm{E}}[Y_i \mid W_i=1, X_i=60] - \mathop{\mathrm{E}}[Y_i \mid W_i=0, X_i=60] \]
The difference in subsample means \(\textcolor{blue}{X=60}\) is an unbiased estimator of this quantity.

\[ \begin{aligned} \widehat{CATE}(60) &=\frac{\textcolor{blue}{70} }{1} - \frac{\textcolor{blue}{60} +\textcolor{blue}{70}+\textcolor{blue}{80}}{3} \\ &= \frac{\textcolor{blue}{70} +\textcolor[RGB]{255,0,0}{60} +\textcolor[RGB]{255,0,0}{70} + \textcolor[RGB]{255,0,0}{80}}{4} - \frac{\textcolor{red}{70} +\textcolor{blue}{60} +\textcolor{blue}{70}+\textcolor{blue}{80}}{4} \\ &= \frac{\textcolor[RGB]{255,0,0}{0} +\textcolor[RGB]{255,0,0}{0} +\textcolor[RGB]{255,0,0}{0} + \textcolor[RGB]{255,0,0}{0}}{4} \end{aligned} \]

Conditional ATE for \(\textcolor{magenta}{X=70}\)

Because we have a randomized experiment within the \(\textcolor{magenta}{X=70}\) group …
The difference in subpopulation means is equal to the average effect of treatment within that group.
We call this the conditional average treatment effect (\(CATE(70)\)). \[ CATE(70) = \mathop{\mathrm{E}}[Y_i(1) - Y_i(0) \mid X_i=0] = \mathop{\mathrm{E}}[Y_i \mid W_i=1, X_i=70] - \mathop{\mathrm{E}}[Y_i \mid W_i=0, X_i=70] \]
The difference in subsample means \(\textcolor{magenta}{X=70}\) is an unbiased estimator of this quantity.

\[ \begin{aligned} \widehat{CATE}(70) &= \frac{\textcolor{magenta}{80} }{1} - \frac{\textcolor{magenta}{70} }{1} \\ &=\frac{\textcolor{magenta}{80} +\textcolor[RGB]{255,0,0}{80} }{2} - \frac{\textcolor[RGB]{255,0,0}{70} + \textcolor{magenta}{70}}{2} \\ &= \frac{\textcolor[RGB]{255,0,0}{10} +\textcolor[RGB]{255,0,0}{10}}{2} \end{aligned} \]

The ATE

To get an aggregate summary of the effect of MHT, we can average the two within-subpopulation effects. \[ \begin{aligned} \text{ATE} &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ Y_i(1) - Y_i(0) \mid X_i } ] \\ &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ Y_i \mid W_i=1, X_i } - \mathop{\mathrm{E}}\qty{ Y_i \mid W_i=0, X_i } ] \\ &= \mathop{\mathrm{E}}\qty[ CATE(X_i) ] \end{aligned} \]
We can estimate this with a weighted average of our two estimates. \[ \begin{aligned} \widehat{ATE} &= \textcolor{blue}{\frac{4}{6}} \widehat{CATE}(60) + \textcolor{magenta}{\frac{2}{6}} \widehat{CATE}(70) \\ &=\textcolor{blue}{\frac{4}{6}} \cdot \left\{\frac{70}{1} - \frac{60+70+80}{3}\right\} + \textcolor{magenta}{\frac{2}{6}} \cdot \left\{\frac{80}{1} - \frac{70}{1}\right\} \\ &= \frac{20}{6} \end{aligned} \]

The ATT and ATC

We can, if we prefer, average the CATE over different distributions of units.
This changes our weights. And therefore gives us different summaries of the effect.

\[ \begin{aligned} \text{ATT} &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ Y_i(1) - Y_i(0) \mid X_i } \mid W_i = 1] \\ &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ Y_i \mid W_i=1, X_i } - \mathop{\mathrm{E}}\qty{ Y_i \mid W_i=0, X_i } \mid W_i=1] \\ &= \mathop{\mathrm{E}}\qty[ CATE(X_i) \mid W_i=1] \\ & \\ \widehat{ATT} &= \textcolor{blue}{\frac{1}{2}} \widehat{CATE}(60) + \textcolor{magenta}{\frac{1}{2}} \widehat{CATE}(70) \\ &=\textcolor{blue}{\frac{1}{2}} \cdot \left\{\frac{70}{1} - \frac{60+70+80}{1}\right\} + \textcolor{magenta}{\frac{1}{2}} \cdot \left\{\frac{80}{1} - \frac{70}{1}\right\} \\ &= \frac{10}{2} \end{aligned} \]

\[ \begin{aligned} \text{ATC} &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ Y_i(1) - Y_i(0) \mid X_i } \mid W_i = 0] \\ &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ Y_i \mid W_i=0, X_i } - \mathop{\mathrm{E}}\qty{ Y_i \mid W_i=0, X_i } \mid W_i=0] \\ &= \mathop{\mathrm{E}}\qty[ CATE(X_i) \mid W_i=0] \\ & \\ \widehat{ATC} &= \textcolor{blue}{\frac{3}{4}} \widehat{CATE}(60) + \textcolor{magenta}{\frac{1}{4}} \widehat{CATE}(70) \\ &=\textcolor{blue}{\frac{3}{4}} \cdot \left\{\frac{70}{1} - \frac{60+70+80}{1}\right\} + \textcolor{magenta}{\frac{1}{4}} \cdot \left\{\frac{80}{1} - \frac{70}{1}\right\} \\ &= \frac{10}{4} \end{aligned} \]

Working With All the Data

This formalizes what we were talking about earlier.
It tells us we can use subpopulation means that we can estimate in place of counterfactual means we (otherwise) cannot.

\[ \begin{aligned} \mathop{\mathrm{E}}[Y(1) \mid W_i=1] &= \mathop{\mathrm{E}}[ \mathop{\mathrm{E}}[Y_i \mid W_i=1, X_i]] = \sum_x \textcolor[RGB]{0,191,196}{p(1,x)\mu(1,x)} \approx \sum_x \textcolor[RGB]{0,191,196}{\hat p(1,x) \hat\mu(1,x)} \\ \mathop{\mathrm{E}}[Y(0) \mid W_i=1] &= \mathop{\mathrm{E}}[ \mathop{\mathrm{E}}[Y_i \mid W_i=0, X_i]] = \sum_x \textcolor[RGB]{0,191,196}{p(1,x)}\textcolor[RGB]{248,118,109}{\mu(0,x)} \approx \sum_x \textcolor[RGB]{0,191,196}{\hat p(1,x)} \textcolor[RGB]{248,118,109}{\hat\mu(0,x)} \\ \text{ATT} &= \sum_{x} \textcolor[RGB]{0,191,196}{p(1,x)}\qty{ \textcolor[RGB]{0,191,196}{\mu(1,x)} - \textcolor[RGB]{248,118,109}{\mu(0,x)} } \approx \sum_{x} \textcolor[RGB]{0,191,196}{\hat p(1,x)}\qty{ \textcolor[RGB]{0,191,196}{\hat\mu(1,x)} - \textcolor[RGB]{248,118,109}{\hat\mu(0,x)} } && \text{ and similarly } \\ \text{ATC} &= \sum_{x} \textcolor[RGB]{248,118,109}{p(0,x)}\qty{ \textcolor[RGB]{0,191,196}{\mu(1,x)} - \textcolor[RGB]{248,118,109}{\mu(0,x)} } \approx \sum_{x} \textcolor[RGB]{0,191,196}{\hat p(1,x)}\qty{ \textcolor[RGB]{0,191,196}{\hat\mu(1,x)} - \textcolor[RGB]{248,118,109}{\hat\mu(0,x)} }. \end{aligned} \]

So the adjusted gap we were talking about last time is the ATC.
Given randomization. And we can calculate an estimate pretty easily.

mu.hat = function(w,x) { mean(Y[X==x & W==w]) }         |> Vectorize()
p.hat  = function(w,x) { sum(X==x & W==w) / sum(W==w) } |> Vectorize()
ATC.hat = sum(p.hat(0,x)*(mu.hat(1,x) - mu.hat(0,x)))
ATT.hat = sum(p.hat(1,x)*(mu.hat(1,x) - mu.hat(0,x)))

The ATT and ATC are 0.00 and 0.00 respectively.